10 Gigabit Ethernet

10 gigabit router

The 10 Gigabit Ethernet or 10GE or 10GbE or 10 GigE standard was first published in 2002 as IEEE Std 802.3ae-2002. It defines a version of Ethernet with a nominal data rate of 10 Gbit/s, ten times as fast as Gigabit Ethernet.

Over the years the following 802.3 standards relating to 10GbE have been published: 802.3ae-2002 (fiber -SR, -LR, -ER and -LX4 PMDs), 802.3ak-2004 (-CX4 copper twin-ax InfiniBand type cable), 802.3an-2006 (10GBASE-T copper twisted pair), 802.3ap-2007 (copper backplane -KR and -KX4 PMDs) and 802.3aq-2006 (fiber -LRM PMD with enhanced equalization).

The 802.3ae-2002 and 802.3ak-2004 amendments were consolidated into the IEEE 802.3-2005 standard. IEEE 802.3-2005 and the other amendments have been consolidated into IEEE Std 802.3-2008.

10 Gigabit Ethernet supports only full duplex links which can be connected by switches. Half duplex operation and CSMA/CD (carrier sense multiple access with collision detect) are not supported in 10GbE.

The 10 Gigabit Ethernet standard encompasses a number of different physical layer (PHY) standards. As of 2008 10 Gigabit Ethernet is still an emerging technology with only 1 million ports shipped in 2007, and it remains to be seen which of the PHYs will gain widespread commercial acceptance. A networking device may support different PHY types by means of pluggable PHY modules.

At the time the 10 Gigabit Ethernet standard was developed there was much interest in 10GbE as a WAN transport and this led to the introduction of the concept of the WAN PHY for 10GbE. This operates at a slightly slower data-rate than the LAN PHY and adds some extra encapsulation. The WAN PHY and LAN PHY are specified to share the same PMDs (physical medium dependent) so 10GBASE-LR and 10GBASE-LW can use the same optics. In terms of number of ports shipped the LAN PHY greatly outsells the WAN PHY.

Optical modules

Optical modules are not specified in 802.3 but by multi-source agreements (MSAs). The relevant MSAs for 10GbE are XENPAK, X2, XPAK, XFP and SFP+.^[1]

When choosing a PHY module, consideration should be given to cost, reach, media type, power consumption and size (form factor).

In the case of SFP+ consideration also has to be given to whether the module is linear or limiting. Linear SFP+ modules are appropriate for 10GBASE-LRM otherwise limiting modules are preferred.^[2]

XENPAK was the first MSA for 10GE and has the largest form factor.

X2 and XPAK were later competing standards with smaller form factors. X2 and XPAK have not been as successful in the market as XENPAK.

XFP came after X2 and XPAK and it is also smaller.

The newest module standard, SFP+ (see SFP transceiver), developed by the ANSI T11 fibre channel group is smaller still and lower power than XFP. SFP+ is now the most popular socket on 10GE systems.^[3]^[4] SFP+ modules do only optical to electrical conversion, no clock and data recovery, putting a higher burden on the host's channel equalization. SFP+ modules share a common physical form factor with legacy SFP modules, allowing higher port density than XFP and the re-use of existing designs for 24 or 48 ports in a 19" rack width blade.

Optical modules are connected to a host by either a XAUI, XFI or SFI interface.

XENPAK, X2, and XPAK modules use XAUI to connect to their hosts. XAUI (XGXS) uses a four-lane data channel and is specified in IEEE 802.3 Clause 48.

XFP modules use a XFI interface and SFP+ modules use a SFI interface. XFI and SFI use a single lane data channel and the encoding specified in IEEE 802.3 Clause 49.

Fiber (10GBASE-R)

A Router with 10 Gigabit Ethernet Optical Interfaces. The yellow cables are for potentially long range, single mode fiber connections.

There are two classifications for optical fiber^[5]: single-mode (SMF) and multi-mode (MMF). In SMF light follows a single path through the fiber while in MMF it takes multiple paths resulting in differential mode delay (DMD). SMF is used for long distance communication and MMF is used for distances of less than 300 m. SMF has a narrower core (8.3 µm) which requires a more precise terminations and connection method. MMF has a wider core (50 or 62.5 µm) and is generally less expensive than SMF. The advantage of MMF for short distances is that because of its wider core it can be driven by lower cost lasers and its connectors are cheaper and more reliable. Its disadvantage is that due to DMD it can work only over short distances.^[6] To distinguish SMF from MMF cables, SMF cables are usually yellow, while MMF cables are orange.^[7]

New structured cabling installations use OM3 50 µm MMF which has no center defect. OM3 cable can carry 10GBE 300 m using low cost 10GBASE-SR optics or use 10GBASE-LX4 without a mode conditioning patch cord. See ISO 11801 and multi mode fibre.

Older installations use FDDI grade 62.5 µm MMF which has a center defect and is harder for the 10GBE optical modules to drive^[8]. For -LX4 a mode conditioning patch cord is needed that adds extra cost to an installation.

10GBASE-SR

10GBASE-SR ("short range") uses the IEEE 802.3 Clause 49 64B/66B Physical Coding Sublayer (PCS) and 850 nm lasers. It delivers serialized data over multi-mode fiber at a line rate of 10.3125 Gbit/s.

Over deployed multi-mode fiber cabling, it has a range of between 26 metres (85 ft) and 82 metres (269 ft) depending on cable type. Over new 50 μm 2000 MHz·km OM3 multi-mode fiber (MMF), it has a range of 300 metres (980 ft).^[9] The transmitter can be implemented with a VCSEL (Vertical Cavity Surface Emitting Laser) which is low cost and low power. MMF has the advantage of having lower cost connectors than SMF because of its wider core. OM3 is now the preferred choice for structured optical cabling within buildings.

10GBASE-SR delivers the lowest cost, lowest power and smallest form factor optical modules.

10GBASE-LR

10GBASE-LR ("long range") uses the IEEE 802.3 Clause 49 64B/66B Physical Coding Sublayer (PCS) and 1310 nm lasers. It delivers serialized data over single-mode fiber at a line rate of 10.3125 Gbit/s.

10GBASE-LR has a specified reach of 10 kilometres (6.2 mi), but 10GBASE-LR optical modules can often manage distances of up to 25 kilometres (16 mi) with no data loss.

Fabry–Pérot lasers are commonly used in 10GBASE-LR optical modules. Fabry–Pérot lasers are more expensive than VCSELs but their high power and focused beam allow efficient coupling into the small core of single mode fiber.

10GBASE-LRM

10GBASE-LRM, (Long Reach Multimode) also known as 802.3aq^[10] uses the IEEE 802.3 Clause 49 64B/66B Physical Coding Sublayer (PCS) and 1310 nm lasers. It delivers serialized data over multi-mode fiber at a line rate of 10.3125 Gbit/s.

10GBASE-LRM supports distances up to 220 metres (720 ft) on FDDI-grade 62.5 µm multi-mode fibre originally installed in the early 1990s for FDDI and 100BaseFX networks and 260 metres (850 ft) on OM3. 10GBASE-LRM reach is not quite as far as the older 10GBASE-LX4 standard. However it is hoped that 10GBASE-LRM modules will be lower cost and lower power; and because 10GBASE-LRM uses the same PCS as XFI and SFI no recoding of data is required with the XFP and SFP+ module MSAs.

10GBASE-LRM uses electronic dispersion control (EDC) for receive equalization^[11]....

10GBASE-ER

10GBASE-ER ("extended range") uses the IEEE 802.3 Clause 49 64B/66B Physical Coding Sublayer (PCS) and 1550 nm lasers. It delivers serialized data over single-mode fiber at a line rate of 10.3125 Gbit/s.

10GBASE-ER has a reach of 40 kilometres (25 mi) ^[12].

10GBASE-ZR

Several manufacturers have introduced 80 km (50 mi) range ER pluggable interfaces under the name 10GBASE-ZR. This 80 km PHY is not specified within the IEEE 802.3ae standard and manufacturers have created their own specifications based upon the 80 km PHY described in the OC-192/STM-64 SDH/SONET specifications.

The 802.3 standard will not be amended to cover the ZR PHY.

10GBASE-LX4

10GBASE-LX4 uses the IEEE 802.3 Clause 48 Physical Coding Sublayer (PCS) and coarse WDM.

It supports ranges of between 240 metres (790 ft) and 300 metres (980 ft) over legacy multi-mode cabling. This is achieved through the use of four separate laser sources operating at 3.125 Gbit/s in the range of 1300 nm on unique wavelengths. 10GBASE-LX4 also supports 10 kilometres (6.2 mi) over SMF.

Until 2005 10GBASE-LX4 optical modules were cheaper than 10GBASE-LR optical modules.

10GBASE-LX4 is used by people who want to support both MMF and SMF with a single optical module. Because 10GBASE-LX4 uses four lasers it has a potential cost, size and power disadvantage compared to 10GBASE-LRM.

When used with legacy MMF an expensive mode conditioning patch cord is needed. The mode conditioning patch cord is a short length of SMF which connects to the MMF in such a way to move the beam away from the central defect in the legacy MMF. This is not needed with OM3.

Copper (10GBASE-X/10GBASE-T)

10G Ethernet can also run over twin-ax and cat 6a cabling as well as backplanes.

10GBASE-CX4

10GBASE-CX4 — was the first 10G copper standard published by 802.3 (as 802.3ak-2004). It uses the XAUI 4-lane PCS (Clause 48) and copper cabling similar to that used by InfiniBand technology. It is specified to work up to a distance of 15 m (49 ft). Each lane carries 3.125 G baud of signaling bandwidth.

10GBASE-CX4 offers the advantages of low power, low cost and low latency, but has a bigger form factor than the newer single lane SFP+ standard and a much shorter reach than fibre or 10GBASE-T.

SFP+ Direct Attach (10GSFP+Cu)

This uses a passive twin-ax cable assembly and connects directly into an SFP+ housing. It has a range of 10 m and like 10GBASE-CX4 is low power, low cost and low latency with the added advantage of having the small form factor of SFP+. SFP+ Direct Attach is expected to be the optimum solution for reaches of 10 m.^[13]

10GBASE-KX4 and 10GBASE-KR

Backplane Ethernet — also known by its working group name 802.3ap — is used in backplane applications such as blade servers and routers/switches with upgradable line cards. 802.3ap implementations are required to operate in an environment comprising up to 1 metre (39 in) of copper printed circuit board with two connectors. The standard provides for two different implementations at 10Gbit/s: 10GBASE-KX4 and 10GBASE-KR. 10GBASE-KX4 uses the same physical layer coding (defined in IEEE 802.3 Clause 48) as 10GBASE-CX4. 10GBASE-KR uses the same coding (defined in IEEE 802.3 Clause 49) as 10GBASE-LR/ER/SR. The 802.3ap standard also defines an optional layer for FEC, a backplane autonegotiation protocol and link training where the receiver can set a three tap transmit equalizer.

10GBASE-T

10GBASE-T, or IEEE 802.3an-2006, is a standard released in 2006 to provide 10 gigabit/second connections over unshielded or shielded twisted pair cables, over distances up to 100 metres (330 ft).^[14] 10GBASE-T cable infrastructure can also be used for 1000BASE-T allowing a gradual upgrade from 1000BASE-T using autonegotiation to select which speed to use. 10GBASE-T has higher latency and consumes more power than other 10 gigabit Ethernet physical layers. In 2008 10GBASE-T silicon is now available from several manufacturers ^[15]^[16]^[17]^[18] with claimed power dissipation of 6W and a latency approaching 1 microsecond ^[19].

Connectors

10GBASE-T uses the venerable IEC 60603-7 8P8C (commonly known as RJ45) connectors already widely used with Ethernet. Transmission characteristics are now specified to 500 MHz.

Cables

Category 6A or better balanced twisted pair cables specified in ISO 11801 amendment 2 or ANSI/TIA-568-C.2 are needed to support 10GBASE-T up to distances of 100 m. Category 6 cables can support 10GBASE-T for shorter distances when qualified according to the guidelines in ISO TR 24750 or TIA-155-A.

Electrical characteristics

The 802.3an standard defines the wire-level modulation for 10GBASE-T as a Tomlinson-Harashima precoded (THP) version of pulse-amplitude modulation with 16 discrete levels (PAM-16), encoded in a two-dimensional checkerboard pattern known as DSQ128. Several proposals were considered for wire-level modulation, including PAM with 12 discrete levels (PAM-12), 10 levels (PAM-10), or 8 levels (PAM-8), both with and without Tomlinson-Harashima Precoding (THP). PAM-5 is what is used in the older 1000BASE-T gigabit Ethernet standard.

WAN PHY (10GBASE-W)

10GBASE-SW, 10GBASE-LW, 10GBASE-EW, and 10GBASE-ZW are varieties that use the WAN PHY, designed to interoperate with OC-192/STM-64 SDH/SONET equipment using a light-weight SDH/SONET frame running at 9.953 Gbit/s. WAN PHY is used when an enterprise user wishes to transport 10G Ethernet across telco SDH/SONET or previously installed WDM systems without having to directly map the Ethernet frames into SDH/SONET. The WAN PHY variants correspond at the physical layer to 10GBASE-SR, 10GBASE-LR, 10GBASE-ER and 10GBASE-ZR respectively, and hence use the same types of fiber and support the same distances. There is no WAN PHY standard corresponding to 10GBASE-LX4 and 10GBASE-CX4 since the original SONET/SDH standard requires a serial implementation. The WAN PHYs use 10GBASE-W as the PCS.

10GbE NICs

10GbE network interface cards are available from several manufacturers. These plug into ordinary computer servers using PCI express, or PCI-X and connect to the LAN with a choice of PHY modules.

40 Gigabit DWDM

A number of vendors are offering products based on Dense wavelength-division multiplexing, which allows packing up to four light carriers into one single-mode optical fiber, effectively providing 40 Gigabit operation. These products are not compatible with the upcoming 40 Gigabit Ethernet standard.